- Okay, let's get started.
Alright, so welcome to lecture five.
Today we're going to be getting to the title of the class,
Convolutional Neural Networks.
Okay, so a couple of administrative details
before we get started.
Assignment one is due Thursday,
April 20, 11:59 p.m. on Canvas.
We're also going to be releasing assignment two on Thursday.
Okay, so a quick review of last time.
We talked about neural networks, and how we had
the running example of the linear score function
that we talked about through the first few lectures.
And then we turned this into a neural network
by stacking these linear layers on top of each other
with non-linearities in between.
And we also saw that this could help address
the mode problem where we are able to learn
intermediate templates that are looking for,
for example, different types of cars, right.
A red car versus a yellow car and so on.
And to combine these together to come up with
the final score function for a class.
Okay, so today we're going to talk about
convolutional neural networks,
which is basically the same sort of idea,
but now we're going to learn convolutional layers
that reason on top of basically explicitly
trying to maintain spatial structure.
So, let's first talk a little bit about
the history of neural networks, and then also
how convolutional neural networks were developed.
So we can go all the way back to 1957 with Frank Rosenblatt,
who developed the Mark I Perceptron machine,
which was the first implementation of an algorithm
called the perceptron, which had sort of the similar idea
of getting score functions, right, using some,
you know, W times X plus a bias.
But here the outputs are going to be either one or a zero.
And then in this case we have an update rule,
so an update rule for our weights, W,
which also look kind of similar to the type of update rule
that we're also seeing in backprop, but in this case
there was no principled backpropagation technique yet,
we just sort of took the weights and adjusted them
in the direction towards the target that we wanted.
So in 1960, we had Widrow and Hoff,
who developed Adaline and Madaline, which was the first time
that we were able to get, to start to stack
these linear layers into multilayer perceptron networks.
And so this is starting to now look kind of like this idea
of neural network layers, but we still didn't have backprop
or any sort of principled way to train this.
And so the first time backprop was really introduced
was in 1986 with Rumelhart.
And so here we can start seeing, you know, these kinds of
equations with the chain rule and the update rules
that we're starting to get familiar with, right,
and so this is the first time we started
to have a principled way to train
these kinds of network architectures.
And so after that, you know, it still wasn't able to scale
to very large neural networks, and so there was sort of
a period in which there wasn't a whole lot
of new things happening here, or a lot of popular use
of these kinds of networks.
And so this really started being reinvigorated
around the 2000s, so in 2006, there was this paper
by Geoff Hinton and Ruslan Salakhutdinov,
which basically showed that we could train
a deep neural network,
and show that we could do this effectively.
But it was still not quite
the sort of modern iteration of neural networks.
It required really careful initialization
in order to be able to do backprop,
and so what they had here was they would have
this first pre-training stage, where you model
each hidden layer through this kind of,
through a restricted Boltzmann machine,
and so you're going to get some initialized weights
by training each of these layers iteratively.
And so once you get all of these hidden layers
you then use that to initialize your, you know,
your full neural network, and then from there
you do backprop and fine tuning of that.
And so when we really started to get the first really strong
results using neural networks, and what sort of really
sparked the whole craze of starting to use these
kinds of networks really widely was at around 2012,
where we had first the strongest results
using for speech recognition,
and so this is work out of Geoff Hinton's lab
for acoustic modeling and speech recognition.
And then for image recognition, 2012 was the landmark paper
from Alex Krizhevsky in Geoff Hinton's lab,
which introduced the first convolutional neural network
architecture that was able to do,
get really strong results on ImageNet classification.
And so it took the ImageNet, image classification benchmark,
and was able to dramatically reduce
the error on that benchmark.
And so since then, you know, ConvNets have gotten
really widely used in all kinds of applications.
So now let's step back and take a look at what gave rise
to convolutional neural networks specifically.
And so we can go back to the 1950s,
where Hubel and Wiesel did a series of experiments
trying to understand how neurons
in the visual cortex worked,
and they studied this specifically for cats.
And so we talked a little bit about this in lecture one,
but basically in these experiments they put electrodes
in the cat, into the cat brain,
and they gave the cat different visual stimulus.
Right, and so, things like, you know,
different kinds of edges, oriented edges,
different sorts of shapes, and they measured
the response of the neurons to these stimuli.
And so there were a couple of important conclusions
that they were able to make, and observations.
And so the first thing found that, you know,
there's sort of this topographical mapping in the cortex.
So nearby cells in the cortex also represent
nearby regions in the visual field.
And so you can see for example, on the right here
where if you take kind of the spatial mapping
and map this onto a visual cortex
there's more peripheral regions are these blue areas,
you know, farther away from the center.
And so they also discovered that these neurons
had a hierarchical organization.
And so if you look at different types of visual stimuli
they were able to find that at the earliest layers
retinal ganglion cells were responsive to things
that looked kind of like circular regions of spots.
And then on top of that there are simple cells,
and these simple cells are responsive to oriented edges,
so different orientation of the light stimulus.
And then going further, they discover that these
were then connected to more complex cells,
which were responsive to both light orientation
as well as movement, and so on.
And you get, you know, increasing complexity,
for example, hypercomplex cells are now responsive
to movement with kind of an endpoint, right,
and so now you're starting to get the idea of corners
and then blobs and so on.
And so
then in 1980, the neocognitron was the first example
of a network architecture, a model,
that had this idea of simple and complex cells
that Hubel and Wiesel had discovered.
And in this case Fukushima put these into
these alternating layers of simple and complex cells,
where you had these simple cells
that had modifiable parameters, and then complex cells
on top of these that performed a sort of pooling
so that it was invariant to, you know,
different minor modifications from the simple cells.
And so this is work that was in the 1980s, right,
and so by 1998 Yann LeCun
basically showed the first example
of applying backpropagation and gradient-based learning
to train convolutional neural networks
that did really well on document recognition.
And specifically they were able to do a good job
of recognizing digits of zip codes.
And so these were then used pretty widely
for zip code recognition in the postal service.
But beyond that it wasn't able to scale yet
to more challenging and complex data, right,
digits are still fairly simple
and a limited set to recognize.
And so this is where Alex Krizhevsky, in 2012,
gave the modern incarnation of convolutional neural networks
and his network we sort of colloquially call AlexNet.
But this network really didn't look so much different
than the convolutional neural networks
that Yann LeCun was dealing with.
They're now, you know, they were scaled now
to be larger and deeper and able to,
the most important parts were that they were now able
to take advantage of the large amount of data
that's now available, in web images, in ImageNet data set.
As well as take advantage
of the parallel computing power in GPUs.
And so we'll talk more about that later.
But fast forwarding today, so now, you know,
ConvNets are used everywhere.
And so we have the initial classification results
on ImageNet from Alex Krizhevsky.
This is able to do a really good job of image retrieval.
You can see that when we're trying to retrieve a flower
for example, the features that are learned
are really powerful for doing similarity matching.
We also have ConvNets that are used for detection.
So we're able to do a really good job of localizing
where in an image is, for example, a bus, or a boat,
and so on, and draw precise bounding boxes around that.
We're able to go even deeper beyond that to do segmentation,
right, and so these are now richer tasks
where we're not looking for just the bounding box
but we're actually going to label every pixel
in the outline of, you know, trees, and people, and so on.
And these kind of algorithms are used in,
for example, self-driving cars,
and a lot of this is powered by GPUs as I mentioned earlier,
that's able to do parallel processing
and able to efficiently train and run these ConvNets.
And so we have modern powerful GPUs as well as ones
that work in embedded systems, for example,
that you would use in a self-driving car.
So we can also look at some of the other applications
that ConvNets are used for.
So, face-recognition, right, we can put an input image
of a face and get out a likelihood of who this person is.
ConvNets are applied to video, and so this is an example
of a video network that looks at both images
as well as temporal information,
and from there is able to classify videos.
We're also able to do pose recognition.
Being able to recognize, you know,
shoulders, elbows, and different joints.
And so here are some images of our fabulous TA, Lane,
in various kinds of pretty non-standard human poses.
But ConvNets are able to do a pretty good job
of pose recognition these days.
They're also used in game playing.
So some of the work in reinforcement learning,
deeper enforcement learning that you may have seen,
playing Atari games, and Go, and so on,
and ConvNets are an important part of all of these.
Some other applications, so they're being used for
interpretation and diagnosis of medical images,
for classification of galaxies, for street sign recognition.
There's also whale recognition,
this is from a recent Kaggle Challenge.
We also have examples of looking at aerial maps
and being able to draw out where are the streets
on these maps, where are buildings,
and being able to segment all of these.
And then beyond recognition of classification detection,
these types of tasks, we also have tasks
like image captioning, where given an image,
we want to write a sentence description
about what's in the image.
And so this is something that we'll go into
a little bit later in the class.
And we also have, you know, really, really fancy and cool
kind of artwork that we can do using neural networks.
And so on the left is an example of a deep dream,
where we're able to take images and kind of hallucinate
different kinds of objects and concepts in the image.
There's also neural style type work, where we take an image
and we're able to re-render this image
using a style of a particular artist and artwork, right.
And so here we can take, for example, Van Gogh on the right,
Starry Night, and use that to redraw
our original image using that style.
And Justin has done a lot of work in this
and so if you guys are interested,
these are images produced by some of his code
and you guys should talk to him more about it.
Okay, so basically, you know, this is just a small sample
of where ConvNets are being used today.
But there's really a huge amount that can be done with this,
right, and so, you know, for you guys' projects,
sort of, you know, let your imagination go wild
and we're excited to see what sorts of applications
you can come up with.
So today we're going to talk about
how convolutional neural networks work.
And again, same as with neural networks, we're going to first
talk about how they work from a functional perspective
without any of the brain analogies.
And then we'll talk briefly about some of these connections.
Okay, so, last lecture, we talked about
this idea of a fully connected layer.
And how, you know, for a fully connected layer
what we're doing is we operate on top of these vectors,
right, and so let's say we have, you know, an image,
a 3D image, 32 by 32 by three,
so some of the images that we were looking at previously.
We'll take that, we'll stretch all of the pixels out, right,
and then we have this 3072 dimensional vector,
for example in this case.
And then we have these weights, right,
so we're going to multiply this by a weight matrix.
And so here for example our W we're going to say is 10 by 3072.
And then we're going to get the activations,
the output of this layer, right, and so in this case,
we take each of our 10 rows and we do this dot product
with 3072 dimensional input.
And from there we get this one number
that's kind of the value of that neuron.
And so in this case we're going to have
10 of these neuron outputs.
And so a convolutional layer, so the main difference
between this and the fully connected layer
that we've been talking about
is that here we want to preserve spatial structure.
And so taking this 32 by 32 by three image
that we had earlier, instead of stretching this all out
into one long vector, we're now going to keep the structure
of this image, right, this three dimensional input.
And then what we're going to do is
our weights are going to be these small filters,
so in this case for example, a five by five by three filter,
and we're going to take this filter
and we're going to slide it over the image spatially
and compute dot products at every spatial location.
And so we're going to go into detail of exactly how this works.
So, our filters, first of all,
always extend the full depth of the input volume.
And so they're going to be just a smaller spatial area,
so in this case five by five, right,
instead of our full 32 by 32 spatial input,
but they're always going to go through the full depth, right,
so here we're going to take five by five by three.
And then we're going to take this filter
and at a given spatial location
we're going to do a dot product
between this filter and then a chunk of a image.
So we're just going to overlay this filter
on top of a spatial location in the image,
right, and then do the dot product,
the multiplication of each element of that filter
with each corresponding element in that spatial location
that we've just plopped it on top of.
And then this is going to give us a dot product.
So in this case, we have five times five times three,
this is the number of multiplications
that we're going to do, right, plus the bias term.
And so this is basically taking our filter W
and basically doing W transpose times X and plus bias.
So is that clear how this works?
Yeah, question.
[faint speaking]
Yeah, so the question is, when we do the dot product
do we turn the five by five by three into one vector?
Yeah, in essence that's what you're doing.
You can, I mean, you can think of it as just
plopping it on and doing the element-wise multiplication
at each location, but this is going to give you the same result
as if you stretched out the filter at that point,
stretched out the input volume that it's laid over,
and then took the dot product,
and that's what's written here, yeah, question.
[faint speaking]
Oh, this is, so the question is,
any intuition for why this is a W transpose?
And this was just, not really,
this is just the notation that we have here
to make the math work out as a dot product.
So it just depends on whether, how you're representing W
and whether in this case if we look at the W matrix
this happens to be each column and so we're just taking
the transpose to get a row out of it.
But there's no intuition here,
we're just taking the filters of W
and we're stretching it out into a one D vector,
and in order for it to be a dot product
it has to be like a one by, one by N vector.
[faint speaking]
Okay, so the question is,
is W here not five by five by three, it's one by 75.
So that's the case, right, if we're going
to do this dot product of W transpose times X,
we have to stretch it out first
before we do the dot product.
So we take the five by five by three,
and we just take all these values
and stretch it out into a long vector.
And so again, similar to the other question,
the actual operation that we're doing here
is plopping our filter on top of
a spatial location in the image
and multiplying all of the corresponding values together,
but in order just to make it kind of an easy expression
similar to what we've seen before
we can also just stretch each of these out,
make sure that dimensions are transposed correctly
so that it works out as a dot product.
Yeah, question.
[faint speaking]
Okay, the question is,
how do we slide the filter over the image.
We'll go into that next, yes.
[faint speaking]
Okay, so the question is, should we rotate the kernel
by 180 degrees to better match the convolution,
the definition of a convolution.
And so the answer is that we'll also show the equation
for this later, but we're using convolution
as kind of a looser definition of what's happening.
So for people from signal processing,
what we are actually technically doing,
if you want to call this a convolution,
is we're convolving with the flipped version of the filter.
But for the most part, we just don't worry about this
and we just, yeah, do this operation
and it's like a convolution in spirit.
Okay, so...
Okay, so we had a question earlier, how do we, you know,
slide this over all the spatial locations.
Right, so what we're going to do is
we're going to take this filter, we're going to start
at the upper left-hand corner and basically center
our filter on top of every pixel in this input volume.
And at every position, we're going to do this dot product
and this will produce one value
in our output activation map.
And so then we're going to just slide this around.
The simplest version is just at every pixel
we're going to do this operation and fill in
the corresponding point in our output activation.
You can see here that the dimensions are not exactly
what would happen, right, if you're going to do this.
I had 32 by 32 in the input
and I'm having 28 by 28 in the output,
and so we'll go into examples later of the math
of exactly how this is going to work out dimension-wise,
but basically you have a choice
of how you're going to slide this,
whether you go at every pixel or whether you slide,
let's say, you know, two input values over at a time,
two pixels over at a time,
and so you can get different size outputs
depending on how you choose to slide.
But you're basically doing this operation in a grid fashion.
Okay, so what we just saw earlier,
this is taking one filter, sliding it over
all of the spatial locations in the image
and then we're going to get this activation map out, right,
which is the value of that filter at every spatial location.
And so when we're dealing with a convolutional layer,
we want to work with multiple filters, right,
because each filter is kind of looking for a specific
type of template or concept in the input volume.
And so we're going to have a set of multiple filters,
and so here I'm going to take a second filter,
this green filter, which is again five by five by three,
I'm going to slide this over all of the spatial locations
in my input volume, and then I'm going to get out
this second green activation map also of the same size.
And we can do this for as many filters
as we want to have in this layer.
So for example, if we have six filters,
six of these five by five filters,
then we're going to get in total six activation maps out.
All of, so we're going to get this output volume
that's going to be basically six by 28 by 28.
Right, and so a preview of how we're going to use
these convolutional layers in our convolutional network
is that our ConvNet is basically going to be
a sequence of these convolutional layers
stacked on top of each other, same way as what we had
with the simple linear layers in their neural network.
And then we're going to intersperse these
with activation functions,
so for example, a ReLU activation function.
Right, and so you're going to get something like Conv, ReLU,
and usually also some pooling layers,
and then you're just going to get a sequence of these
each creating an output that's now going to be
the input to the next convolutional layer.
Okay, and so each of these layers, as I said earlier,
has multiple filters, right, many filters.
And each of the filter is producing an activation map.
And so when you look at multiple of these layers
stacked together in a ConvNet, what ends up happening
is you end up learning this hierarching of filters,
where the filters at the earlier layers usually represent
low-level features that you're looking for.
So things kind of like edges, right.
And then at the mid-level,
you're going to get more complex kinds of features,
so maybe it's looking more for things
like corners and blobs and so on.
And then at higher-level features,
you're going to get things that are starting
to more resemble concepts than blobs.
And we'll go into more detail later in the class
in how you can actually visualize all these features
and try and interpret what your network,
what kinds of features your network is learning.
But the important thing for now is just to understand
that what these features end up being
when you have a whole stack of these,
is these types of simple to more complex features.
[faint speaking]
Yeah.
Oh, okay.
Oh, okay, so the question is, what's the intuition
for increasing the depth each time.
So here I had three filters in the original layer
and then six filters in the next layer.
Right, and so this is mostly a design choice.
You know, people in practice have found
certain types of these configurations to work better.
And so later on we'll go into case studies of different
kinds of convolutional neural network architectures
and design choices for these
and why certain ones work better than others.
But yeah, basically the choice of,
you're going to have many design choices
in a convolutional neural network,
the size of your filter, the stride,
how many filters you have,
and so we'll talk about this all more later.
Question.
[faint speaking]
Yeah, so the question is, as we're sliding this filter
over the image spatially it looks like we're sampling
the edges and corners less than the other locations.
Yeah, that's a really good point,
and we'll talk I think in a few slides
about how we try and compensate for that.
Okay, so each of these convolutional layers
that we have stacked together, we saw how we're starting
with more simpler features and then aggregating these
into more complex features later on.
And so in practice this is compatible
with what Hubel and Wiesel noticed in their experiments,
right, that we had these simple cells
at the earlier stages of processing,
followed by more complex cells later on.
And so even though we didn't explicitly
force our ConvNet to learn these kinds of features,
in practice when you give it this type of
hierarchical structure and train it using backpropagation,
these are the kinds of filters that end up being learned.
[faint speaking]
Okay, so yeah, so the question is,
what are we seeing in these visualizations.
And so, alright so, in these visualizations, like,
if we look at this Conv1, the first convolutional layer,
each of these grid, each part of this grid is a one neuron.
And so what we've visualized here
is what the input looks like that maximizes
the activation of that particular neuron.
So what sort of image you would get
that would give you the largest value,
make that neuron fire and have the largest value.
And so the way we do this is basically
by doing backpropagation from a particular neuron activation
and seeing what in the input will trigger,
will give you the highest values of this neuron.
And this is something that we'll talk about
in much more depth in a later lecture
about how we create all of these visualizations.
But basically each element of these grids
is showing what in the input would look like
that basically maximizes the activation of the neuron.
So in a sense, what is the neuron looking for?
Okay, so here is an example of some of the activation maps
produced by each filter, right.
So we can visualize up here on the top
we have this whole row of example five by five filters,
and so this is basically a real case from a trained ConvNet
where each of these is what a five by five filter
looks like, and then as we convolve this over an image,
so in this case this I think it's like a corner of a car,
the car light, what the activation looks like.
Right, and so here for example,
if we look at this first one, this red filter,
filter like with a red box around it,
we'll see that it's looking for,
the template looks like an edge, right, an oriented edge.
And so if you slide it over the image,
it'll have a high value, a more white value
where there are edges in this type of orientation.
And so each of these activation maps is kind of the output
of sliding one of these filters over
and where these filters are causing, you know,
where this sort of template is more present in the image.
And so the reason we call these convolutional is because
this is related to the convolution of two signals,
and so someone pointed out earlier
that this is basically this convolution equation over here,
for people who have seen convolutions before
in signal processing, and in practice
it's actually more like a correlation
where we're convolving with the flipped version
of the filter, but this is kind of a subtlety,
it's not really important for the purposes of this class.
But basically if you're writing out what you're doing,
it has an expression that looks something like this,
which is the standard definition of a convolution.
But this is basically just taking a filter,
sliding it spatially over the image
and computing the dot product at every location.
Okay, so you know, as I had mentioned earlier,
like what our total convolutional neural network
is going to look like is we're going to have an input image,
and then we're going to pass it through
this sequence of layers, right,
where we're going to have a convolutional layer first.
We usually have our non-linear layer after that.
So ReLU is something that's very commonly used
that we're going to talk about more later.
And then we have these Conv, ReLU, Conv, ReLU layers,
and then once in a while we'll use a pooling layer
that we'll talk about later as well
that basically downsamples the size of our activation maps.
And then finally at the end of this we'll take our last
convolutional layer output and then we're going to use
a fully connected layer that we've seen before,
connected to all of these convolutional outputs,
and use that to get a final score function
basically like what we've already been working with.
Okay, so now let's work out some examples
of how the spatial dimensions work out.
So let's take our 32 by 32 by three image as before,
right, and we have our five by five by three filter
that we're going to slide over this image.
And we're going to see how we're going to use that
to produce exactly this 28 by 28 activation map.
So let's assume that we actually have a seven by seven input
just to be simpler, and let's assume
we have a three by three filter.
So what we're going to do is
we're going to take this filter,
plop it down in our upper left-hand corner,
right, and we're going to multiply, do the dot product,
multiply all these values together to get our first value,
and this is going to go into the upper left-hand value
of our activation map.
Right, and then what we're going to do next
is we're just going to take this filter,
slide it one position to the right,
and then we're going to get another value out from here.
And so we can continue with this to have another value,
another, and in the end what we're going to get
is a five by five output, right,
because what fit was basically sliding this filter
a total of five spatial locations horizontally
and five spatial locations vertically.
Okay, so as I said before
there's different kinds of design choices that we can make.
Right, so previously I slid it at every single
spatial location and the interval at which I slide
I'm going to call the stride.
And so previously we used the stride of one.
And so now let's see what happens
if we have a stride of two.
Right, so now we're going to take our first location
the same as before, and then we're going to skip
this time two pixels over and we're going to get
our next value centered at this location.
Right, and so now if we use a stride of two,
we have in total three of these that can fit,
and so we're going to get a three by three output.
Okay, and so what happens when we have a stride of three,
what's the output size of this?
And so in this case, right, we have three,
we slide it over by three again,
and the problem is that here it actually doesn't fit.
Right, so we slide it over by three
and now it doesn't fit nicely within the image.
And so what we in practice we just, it just doesn't work.
We don't do convolutions like this
because it's going to lead to asymmetric outputs happening.
Right, and so just kind of looking at the way
that we computed how many, what the output size is going to be,
this actually can work into a nice formula
where we take our dimension of our input N,
we have our filter size F, we have our stride
at which we're sliding along, and our final output size,
the spatial dimension of each output size
is going to be N minus F divided by the stride plus one,
right, and you can kind of see this as a, you know,
if I'm going to take my filter, let's say I fill it in
at the very last possible position that it can be in
and then take all the pixels before that,
how many instances of moving by this stride can I fit in.
Right, and so that's how this equation kind of works out.
And so as we saw before, right, if we have N equal seven
and F equals three, if we want a stride of one
we plug it into this formula, we get five by five
as we had before, and the same thing we had for two.
And with a stride of three, this doesn't really work out.
And so in practice it's actually common
to zero pad the borders in order to make
the size work out to what we want it to.
And so this is kind of related to a question earlier,
which is what do we do, right, at the corners.
And so what in practice happens is
we're going to actually pad our input image with zeros
and so now you're going to be able to place a filter
centered at the upper right-hand pixel location
of your actual input image.
Okay, so here's a question, so who can tell me
if I have my same input, seven by seven,
three by three filter, stride one,
but now I pad with a one pixel border,
what's the size of my output going to be?
[faint speaking]
So, I heard some sixes, heard some sev,
so remember we have this formula that we had before.
So if we plug in N is equal to seven, F is equal to three,
right, and then our stride is equal to one.
So what we actually get, so actually this is giving us
seven, four, so seven minus three is four,
divided by one plus one is five.
And so this is what we had before.
So we actually need to adjust this formula a little bit,
right, so this was actually, this formula is the case
where we don't have zero padded pixels.
But if we do pad it, then if you now take your new output
and you slide it along,
you'll see that actually seven of the filters fit,
so you get a seven by seven output.
And plugging in our original formula, right,
so our N now is not seven, it's nine,
so if we go back here we have N equals nine
minus a filter size of three, which gives six.
Right, divided by our stride, which is one,
and so still six, and then plus one we get seven.
Right, and so once you've padded it
you want to incorporate this padding into your formula.
Yes, question.
[faint speaking]
Seven, okay, so the question is,
what's the actual output of the size,
is it seven by seven or seven by seven by three?
The output is going to be seven by seven
by the number of filters that you have.
So remember each filter is going to do a dot product
through the entire depth of your input volume.
But then that's going to produce one number, right,
so each filter is, let's see if we can go back here.
Each filter is producing a one by seven by seven
in this case activation map output, and so the depth
is going to be the number of filters that we have.
[faint speaking]
Sorry, let me just, one second go back.
Okay, can you repeat your question again?
[muffled speaking]
Okay, so the question is, how does this connect to before
when we had a 32 by 32 by three input, right.
So our input had depth and here in this example
I'm showing a 2D example with no depth.
And so yeah, I'm showing this for simplicity
but in practice you're going to take your,
you're going to multiply throughout the entire depth
as we had before, so you're going to,
your filter is going to be in this case a three be three
spatial filter by whatever input depth that you had.
So three by three by three in this case.
Yeah, everything else stays the same.
Yes, question.
[muffled speaking]
Yeah, so the question is, does the zero padding
add some sort of extraneous features at the corners?
And yeah, so I mean, we're doing our best to still,
get some value and do, like,
process that region of the image,
and so zero padding is kind of one way to do this,
where I guess we can, we are detecting
part of this template in this region.
There's also other ways to do this that, you know,
you can try and like, mirror the values here
or extend them, and so it doesn't have to be zero padding,
but in practice this is one thing that works reasonably.
And so, yeah, so there is a little bit of kind of artifacts
at the edge and we sort of just,
you do your best to deal with it.
And in practice this works reasonably.
I think there was another question.
Yeah, question.
[faint speaking]
So if we have non-square images, do we ever use a stride
that's different horizontally and vertically?
So, I mean, there's nothing stopping you from doing that,
you could, but in practice we just usually
take the same stride, we usually operate square regions
and we just, yeah we usually just
take the same stride everywhere and it's sort of like,
in a sense it's a little bit like,
it's a little bit like the resolution at which you're,
you know, looking at this image,
and so usually there's kind of, you might want to match
sort of your horizontal and vertical resolutions.
But, yeah, so in practice you could
but really people don't do that.
Okay, another question.
[faint speaking]
So the question is, why do we do zero padding?
So the way we do zero padding
is to maintain the same input size as we had before.
Right, so we started with seven by seven,
and if we looked at just starting your filter
from the upper left-hand corner, filling everything in,
right, then we get a smaller size output,
but we would like to maintain our full size output.
Okay, so,
yeah, so we saw how padding can basically help you
maintain the size of the output that you want,
as well as apply your filter at these,
like, corner regions and edge regions.
And so in general in terms of choosing,
you know, your stride, your filter, your filter size,
your stride size, zero padding, what's common to see
is filters of size three by three, five by five,
seven by seven, these are pretty common filter sizes.
And so each of these, for three by three
you will want to zero pad with one
in order to maintain the same spatial size.
If you're going to do five by five,
you can work out the math, but it's going to come out
to you want to zero pad by two.
And then for seven you want to zero pad by three.
Okay, and so again you know, the motivation
for doing this type of zero padding
and trying to maintain the input size, right,
so we kind of alluded to this before,
but if you have multiple of these layers stacked together...
So if you have multiple of these layers stacked together
you'll see that, you know, if we don't do this kind of
zero padding, or any kind of padding,
we're going to really quickly shrink the size
of the outputs that we have.
Right, and so this is not something that we want.
Like, you can imagine if you have a pretty deep network
then very quickly your, the size of your activation maps
is going to shrink to something very small.
And this is bad both because we're kind of losing out
on some of this information, right,
now you're using a much smaller number of values
in order to represent your original image,
so you don't want that.
And then at the same time also as
we talked about this earlier, your also kind of
losing sort of some of this edge information,
corner information that each time
we're losing out and shrinking that further.
Okay, so let's go through a couple more examples
of computing some of these sizes.
So let's say that we have an input volume
which is 32 by 32 by three.
And here we have 10 five by five filters.
Let's use stride one and pad two.
And so who can tell me
what's the output volume size of this?
So you can think about the formula earlier.
Sorry, what was it?
[faint speaking]
32 by 32 by 10, yes that's correct.
And so the way we can see this, right,
is so we have our input size, F is 32.
Then in this case we want to augment it
by the padding that we added onto this.
So we padded it two in each dimension, right,
so we're actually going to get, total width and total height's
going to be 32 plus four on each side.
And then minus our filter size five,
divided by one plus one and we get 32.
So our output is going to be 32 by 32 for each filter.
And then we have 10 filters total,
so we have 10 of these activation maps,
and our total output volume is going to be 32 by 32 by 10.
Okay, next question,
so what's the number of parameters in this layer?
So remember we have 10 five by five filters.
[faint speaking]
I kind of heard something, but it was quiet.
Can you guys speak up?
250, okay so I heard 250, which is close,
but remember that we're also, our input volume,
each of these filters goes through by depth.
So maybe this wasn't clearly written here
because each of the filters is five by five spatially,
but implicitly we also have the depth in here, right.
It's going to go through the whole volume.
So I heard, yeah, 750 I heard.
Almost there, this is kind of a trick question
'cause also remember we usually always have
a bias term, right, so in practice each filter
has five by five by three weights, plus our one bias term,
we have 76 parameters per filter,
and then we have 10 of these total,
and so there's 760 total parameters.
Okay, and so here's just a summary
of the convolutional layer that you guys can read
a little bit more carefully later on.
But we have our input volume of a certain dimension,
we have all of these choice, we have our filters, right,
where we have number of filters, the filter size,
the stride of the size, the amount of zero padding,
and you basically can use all of these,
go through the computations that we talked about earlier
in order to find out what your output volume is actually
going to be and how many total parameters that you have.
And so some common settings of this.
You know, we talked earlier about common filter sizes
of three by three, five by five.
Stride is usually one and two is pretty common.
And then your padding P is going to be whatever fits,
like, whatever will preserve your spatial extent
is what's common.
And then the total number of filters K,
usually we use powers of two just to be nice, so, you know,
32, 64, 128 and so on, 512,
these are pretty common numbers that you'll see.
And just as an aside,
we can also do a one by one convolution,
this still makes perfect sense where
given a one by one convolution
we still slide it over each spatial extent,
but now, you know, the spatial region
is not really five by five
it's just kind of the trivial case of one by one,
but we are still having this filter
go through the entire depth.
Right, so this is going to be a dot product
through the entire depth of your input volume.
And so the output here, right, if we have an input volume
of 56 by 56 by 64 depth and we're going to do one by one
convolution with 32 filters, then our output is going to be
56 by 56 by our number of filters, 32.
Okay, and so here's an example of a convolutional layer
in TORCH, a deep learning framework.
And so you'll see that, you know, last lecture
we talked about how you can go into these
deep learning frameworks, you can see these definitions
of each layer, right, where they have kind of
the forward pass and the backward pass
implemented for each layer.
And so you'll see convolutions,
spatial convolution is going to be just one of these,
and then the arguments that it's going to take
are going to be all of these design choices of, you know,
I mean, I guess your input and output sizes,
but also your choices of like your kernel width,
your kernel size, padding, and these kinds of things.
Right, and so if we look at another framework, Caffe,
you'll see something very similar,
where again now when you're defining your network
you define networks in Caffe using this kind of, you know,
proto text file where you're specifying
each of your design choices for your layer
and you can see for a convolutional layer
will say things like, you know, the number of outputs
that we have, this is going to be the number of filters
for Caffe, as well as the kernel size and stride and so on.
Okay, and so I guess before I go on,
any questions about convolution,
how the convolution operation works?
Yes, question.
[faint speaking]
Yeah, so the question is,
what's the intuition behind how you choose your stride.
And so at one sense it's kind of the resolution
at which you slide it on, and usually the reason behind this
is because when we have a larger stride
what we end up getting as the output
is a down sampled image, right,
and so what this downsampled image lets us have is both,
it's a way, it's kind of like pooling in a sense
but it's just a different and sometimes works better
way of doing pooling is one of the intuitions behind this,
'cause you get the same effect of downsampling your image,
and then also as you're doing this you're reducing the size
of the activation maps that you're dealing with
at each layer, right, and so this also affects later on
the total number of parameters that you have
because for example at the end of all your Conv layers,
now you might put on fully connected layers on top,
for example, and now the fully connected layer's
going to be connected to every value
of your convolutional output, right,
and so a smaller one will give you smaller number
of parameters, and so now you can get into, like,
basically thinking about trade offs of, you know,
number of parameters you have, the size of your model,
overfitting, things like that, and so yeah,
these are kind of some of the things
that you want to think about with choosing your stride.
Okay, so now if we look a little bit at kind of the,
you know, brain neuron view of a convolutional layer,
similar to what we looked at for the neurons
in the last lecture.
So what we have is that at every spatial location,
we take a dot product between a filter
and a specific part of the image, right,
and we get one number out from here.
And so this is the same idea
of doing these types of dot products, right,
taking your input, weighting it by these Ws, right,
values of your filter, these weights that are the synapses,
and getting a value out.
But the main difference here is just that now
your neuron has local connectivity.
So instead of being connected to the entire input,
it's just looking at a local region spatially of your image.
And so this looks at a local region
and then now you're going to get kind of, you know,
this, how much this neuron is being triggered
at every spatial location in your image.
Right, so now you preserve the spatial structure
and you can say, you know, be able to reason
on top of these kinds of activation maps in later layers.
And just a little bit of terminology,
again for, you know, we have this five by five filter,
we can also call this a five by five receptive field
for the neuron, because this is,
the receptive field is basically the, you know,
input field that this field of vision
that this neuron is receiving, right,
and so that's just another common term
that you'll hear for this.
And then again remember each of these five by five filters
we're sliding them over the spatial locations
but they're the same set of weights,
they share the same parameters.
Okay, and so, you know, as we talked about
what we're going to get at this output
is going to be this volume, right,
where spatially we have, you know, let's say 28 by 28
and then our number of filters is the depth.
And so for example with five filters,
what we're going to get out is this 3D grid
that's 28 by 28 by five.
And so if you look at the filters across
in one spatial location of the activation volume
and going through depth these five neurons,
all of these neurons,
basically the way you can interpret this
is they're all looking at the same region
in the input volume,
but they're just looking for different things, right.
So they're different filters
applied to the same spatial location in the image.
And so just a reminder again kind of comparing
with the fully connected layer that we talked about earlier.
In that case, right, if we look at each of the neurons
in our activation or output, each of the neurons
was connected to the entire stretched out input,
so it looked at the entire full input volume,
compared to now where each one
just looks at this local spatial region.
Question.
[muffled talking]
Okay, so the question is, within a given layer,
are the filters completely symmetric?
So what do you mean by symmetric exactly, I guess?
Right, so okay, so the filters, are the filters doing,
they're doing the same dimension, the same calculation, yes.
Okay, so is there anything different
other than they have the same parameter values?
No, so you're exactly right,
we're just taking a filter with a given set of, you know,
five by five by three parameter values,
and we just slide this in exactly the same way
over the entire input volume to get an activation map.
Okay, so you know, we've gone into a lot of detail
in what these convolutional layers look like,
and so now I'm just going to go briefly
through the other layers that we have
that form this entire convolutional network.
Right, so remember again, we have convolutional layers
interspersed with pooling layers once in a while
as well as these non-linearities.
Okay, so what the pooling layers do
is that they make the representations
smaller and more manageable, right,
so we talked about this earlier with
someone asked a question of why we would want to make
the representation smaller.
And so this is again for it to have fewer,
it effects the number of parameters that you have at the end
as well as basically does some, you know,
invariance over a given region.
And so what the pooling layer does
is it does exactly just downsamples,
and it takes your input volume, so for example,
224 by 224 by 64, and spatially downsamples this.
So in the end you'll get out 112 by 112.
And it's important to note this doesn't do anything
in the depth, right, we're only pooling spatially.
So the number of, your input depth
is going to be the same as your output depth.
And so, for example, a common way to do this is max pooling.
So in this case our pooling layer also has a filter size
and this filter size is going to be the region
at which we pool over, right, so in this case
if we have two by two filters, we're going to slide this,
and so, here, we also have stride two in this case,
so we're going to take this filter
and we're going to slide it along our input volume
in exactly the same way as we did for convolution.
But here instead of doing these dot products,
we just take the maximum value
of the input volume in that region.
Right, so here if we look at the red values,
the value of that will be six is the largest.
If we look at the greens it's going to give an eight,
and then we have a three and a four.
Yes, question.
[muffled speaking]
Yeah, so the question is, is it typical to set up the stride
so that there isn't an overlap?
And yeah, so for the pooling layers it is,
I think the more common thing to do
is to have them not have any overlap,
and I guess the way you can think about this
is basically we just want to downsample
and so it makes sense to kind of look at this region
and just get one value to represent this region
and then just look at the next region and so on.
Yeah, question.
[faint speaking]
Okay, so the question is, why is max pooling
better than just taking the,
doing something like average pooling?
Yes, that's a good point, like, average pooling
is also something that you can do,
and intuition behind why max pooling is commonly used
is that it can have this interpretation of,
you know, if this is, these are activations of my neurons,
right, and so each value is kind of
how much this neuron fired in this location,
how much this filter fired in this location.
And so you can think of max pooling as saying,
you know, giving a signal of how much did this filter fire
at any location in this image.
Right, and if we're thinking about detecting,
you know, doing recognition,
this might make some intuitive sense where you're saying,
well, you know, whether a light or whether some aspect
of your image that you're looking for,
whether it happens anywhere in this region
we want to fire at with a high value.
Question.
[muffled speaking]
Yeah, so the question is, since pooling and stride
both have the same effect of downsampling,
can you just use stride instead of pooling and so on?
Yeah, and so in practice I think
looking at more recent neural network architectures
people have begun to use stride more
in order to do the downsampling instead of just pooling.
And I think this gets into things like, you know,
also like fractional strides and things that you can do.
But in practice this in a sense maybe has a little bit
better way to get better results using that, so.
Yeah, so I think using stride is definitely,
you can do it and people are doing it.
Okay, so let's see, where were we.
Okay, so yeah, so with these pooling layers,
so again, there's right, some design choices that you make,
you take this input volume of W by H by D,
and then you're going to set your hyperparameters
for design choices of your filter size
or the spatial extent over which you are pooling,
as well as your stride, and then you can again compute
your output volume using the same equation that you used
earlier for convolution, it still applies here, right,
so we still have our W total extent
minus filter size divided by stride plus one.
Okay, and so just one other thing to note,
it's also, typically people don't really use zero padding
for the pooling layers because you're just trying
to do a direct downsampling, right,
so there isn't this problem of like,
applying a filter at the corner
and having some part of the filter go off your input volume.
And so for pooling we don't usually have to worry about this
and we just directly downsample.
And so some common settings for the pooling layer
is a filter size of two by two or three by three strides.
Two by two, you know, you can have,
also you can still have pooling of two by two
even with a filter size of three by three,
I think someone asked that earlier,
but in practice it's pretty common just to have two by two.
Okay, so now we've talked about these convolutional layers,
the ReLU layers were the same as what we had before
with the, you know, just the base neural network
that we talked about last lecture.
So we intersperse these and then we have a pooling layer
every once in a while when we feel like downsampling, right.
And then the last thing is that at the end
we want to have a fully connected layer.
And so this will be just exactly the same
as the fully connected layers that you've seen before.
So in this case now what we do
is we take the convolutional network output,
at the last layer we have some volume,
so we're going to have width by height by some depth,
and we just take all of these
and we essentially just stretch these out, right.
And so now we're going to get the same kind of,
you know, basically 1D input that we're used to
for a vanilla neural network, and then we're going to apply
this fully connected layer on top,
so now we're going to have connections
to every one of these convolutional map outputs.
And so what you can think of this is basically,
now instead of preserving, you know,
before we were preserving spatial structure,
right, and so but at the last layer at the end,
we want to aggregate all of this together
and we want to reason basically on top of
all of this as we had before.
And so what you get from that is just our
score outputs as we had earlier.
Okay, so--
- [Student] This is sort of a silly question
about this visual.
Like what are the 16 pixels that are on the far right,
like what should be interpreting those as?
- Okay, so the question is, what are the 16 pixels
that are on the far right, do you mean the--
- [Student] Like that column of--
- [Instructor] Oh, each column.
- [Student] The column on the far right, yeah.
- [Instructor] The green ones or the black ones?
- [Student] The ones labeled pool.
- The one with hold on, pool.
Oh, okay, yeah, so the question is
how do we interpret this column, right, for example at pool.
And so what we're showing here is each of these columns
is the output activation maps, right,
the output from one of these layers.
And so starting from the beginning, we have our car,
after the convolutional layer
we now have these activation maps of each of the filters
slid spatially over the input image.
Then we pass that through a ReLU,
so you can see the values coming out from there.
And then going all the way over,
and so what you get for the pooling layer
is that it's really just taking
the output of the ReLU layer
that came just before it and then it's pooling it.
So it's going to downsample it,
right, and then it's going to take
the max value in each filter location.
And so now if you look at this pool layer output,
like, for example, the last one that you were mentioning,
it looks the same as this ReLU output
except that it's downsampled and that it has this kind of
max value at every spatial location
and so that's the minor difference
that you'll see between those two.
[distant speaking]
So the question is, now this looks like
just a very small amount of information, right,
so how can it know to classify it from here?
And so the way that you should think about this
is that each of these values
inside one of these pool outputs is actually,
it's the accumulation of all the processing that you've done
throughout this entire network, right.
So it's at the very top of your hierarchy,
and so each actually represents
kind of a higher level concept.
So we saw before, you know, for example, Hubel and Wiesel
and building up these hierarchical filters,
where at the bottom level we're looking for edges, right,
or things like very simple structures, like edges.
And so after your convolutional layer
the outputs that you see here in this first column
is basically how much do specific, for example, edges,
fire at different locations in the image.
But then as you go through you're going to get more complex,
it's looking for more complex things, right,
and so the next convolutional layer
is going to fire at how much, you know,
let's say certain kinds of corners show up in the image,
right, because it's reasoning.
Its input is not the original image,
its input is the output, it's already the edge maps, right,
so it's reasoning on top of edge maps,
and so that allows it to get more complex,
detect more complex things.
And so by the time you get all the way up
to this last pooling layer, each value is representing
how much a relatively complex sort of template is firing.
Right, and so because of that now you can just have
a fully connected layer, you're just aggregating
all of this information together to get,
you know, a score for your class.
So each of these values is how much
a pretty complicated complex concept is firing.
Question.
[faint speaking]
So the question is, when do you know you've done
enough pooling to do the classification?
And the answer is you just try and see.
So in practice, you know, these are all design choices
and you can think about this a little bit intuitively,
right, like you want to pool but if you pool too much
you're going to have very few values
representing your entire image and so on,
so it's just kind of a trade off.
Something reasonable versus people have tried
a lot of different configurations
so you'll probably cross validate, right,
and try over different pooling sizes,
different filter sizes, different number of layers,
and see what works best for your problem because yeah,
like every problem with different data is going to,
you know, different set of these sorts
of hyperparameters might work best.
Okay, so last thing, just wanted to point you guys
to this demo of training a ConvNet,
which was created by Andre Karpathy,
the originator of this class.
And so he wrote up this demo
where you can basically train a ConvNet on CIFAR-10,
the dataset that we've seen before, right, with 10 classes.
And what's nice about this demo is you can,
it basically plots for you what each of these filters
look like, what the activation maps look like.
So some of the images I showed earlier
were taken from this demo.
And so you can go try it out, play around with it,
and you know, just go through and try and get a sense
for what these activation maps look like.
And just one thing to note,
usually the first layer activation maps are,
you can interpret them, right,
because they're operating directly on the input image
so you can see what these templates mean.
As you get to higher level layers
it starts getting really hard,
like how do you actually interpret what do these mean.
So for the most part it's just hard to interpret
so you shouldn't, you know, don't worry
if you can't really make sense of what's going on.
But it's still nice just to see the entire flow
and what outputs are coming out.
Okay, so in summary, so today we talked about
how convolutional neural networks work,
how they're basically stacks
of these convolutional and pooling layers
followed by fully connected layers at the end.
There's been a trend towards having smaller filters
and deeper architectures, so we'll talk more
about case studies for some of these later on.
There's also been a trend towards getting rid of these
pooling and fully connected layers entirely.
So just keeping these, just having, you know, Conv layers,
very deep networks of Conv layers,
so again we'll discuss all of this later on.
And then typical architectures again look like this,
you know, as we had earlier.
Conv, ReLU for some N number of steps
followed by a pool every once in a while,
this whole thing repeated some number of times,
and then followed by fully connected ReLU layers
that we saw earlier, you know, one or two
or just a few of these,
and then a softmax at the end for your class scores.
And so, you know, some typical values
you might have N up to five of these.
You're going to have pretty deep layers
of Conv, ReLU, pool sequences, and then usually
just a couple of these fully connected layers at the end.
But we'll also go into some newer architectures
like ResNet and GoogLeNet, which challenge this
and will give pretty different types of architectures.
Okay, thank you and see you guys next time.